Document session replay for multimodal applications

ABSTRACT

Methods, apparatus, and computer program products are described for document session replay for multimodal applications. including identifying, by a multimodal browser in dependence upon a log produced by a Form Interpretation Algorithm (‘FIA’) during a previous document session with a user, a speech prompt provided by a multimodal application in the previous document session; identifying, by a multimodal browser in replay mode in dependence upon the log, a response to the prompt provided by a user of the multimodal application in the previous document session; retrieving, by the multimodal browser in dependence upon the log, an X+V page of the multimodal application associated with the speech prompt and the response; rendering, by the multimodal browser, the visual elements of the retrieved X+V page; replaying, by the multimodal browser, the speech prompt; and replaying, by a multimodal browser, the response.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The field of the invention is data processing, or, more specifically,methods, apparatus, and products for document session replay formultimodal applications.

2. Description of Related Art

User interaction with applications running on small devices through akeyboard or stylus has become increasingly limited and cumbersome asthose devices have become increasingly smaller. In particular, smallhandheld devices like mobile phones and PDAs serve many functions andcontain sufficient processing power to support user interaction throughmultimodal access, that is, by interaction in non-voice modes as well asvoice mode. Devices which support multimodal access combine multipleuser input modes or channels in the same interaction allowing a user tointeract with the applications on the device simultaneously throughmultiple input modes or channels. The methods of input include speechrecognition, keyboard, touch screen, stylus, mouse, handwriting, andothers. Multimodal input often makes using a small device easier.

Multimodal applications are often formed by sets of markup documentsserved up by web servers for display on multimodal browsers. A‘multimodal browser,’ as the term is used in this specification,generally means a web browser capable of receiving multimodal input andinteracting with users with multimodal output, where modes of themultimodal input and output include at least a speech mode. Multimodalbrowsers typically render web pages written in XHTML+Voice (‘X+V’). X+Vprovides a markup language that enables users to interact with anmultimodal application often running on a server through spoken dialogin addition to traditional means of input such as keyboard strokes andmouse pointer action. Visual markup tells a multimodal browser what theuser interface is look like and how it is to behave when the user types,points, or clicks. Similarly, voice markup tells a multimodal browserwhat to do when the user speaks to it. For visual markup, the multimodalbrowser uses a graphics engine; for voice markup, the multimodal browseruses a speech engine. X+V adds spoken interaction to standard webcontent by integrating XHTML (eXtensible Hypertext Markup Language) andspeech recognition vocabularies supported by VoiceXML. For visualmarkup, X+V includes the XHTML standard. For voice markup, X+V includesa subset of VoiceXML. For synchronizing the VoiceXML elements withcorresponding visual interface elements, X+V uses events. XHTML includesvoice modules that support speech synthesis, speech dialogs, command andcontrol, and speech grammars. Voice handlers can be attached to XHTMLelements and respond to specific events. Voice interaction features areintegrated with XHTML and can consequently be used directly within XHTMLcontent.

In addition to X+V, multimodal applications also may be implemented withSpeech Application Tags (‘SALT’). SALT is a markup language developed bythe Salt Forum. Both X+V and SALT are markup languages for creatingapplications that use voice input/speech recognition and voiceoutput/speech synthesis. Both SALT applications and X+V applications useunderlying speech recognition and synthesis technologies or ‘speechengines’ to do the work of recognizing and generating human speech. Asmarkup languages, both X+V and SALT provide markup-based programmingenvironments for using speech engines in an application's userinterface. Both languages have language elements, markup tags, thatspecify what the speech-recognition engine should listen for and whatthe synthesis engine should ‘say.’ Whereas X+V combines XHTML, VoiceXML,and the XML Events standard to create multimodal applications, SALT doesnot provide a standard visual markup language or eventing model. Rather,it is a low-level set of tags for specifying voice interaction that canbe embedded into other environments. In addition to X+V and SALT,multimodal applications may be implemented in Java with a Java speechframework, in C++, for example, and with other technologies and in otherenvironments as well.

Multimodal applications are often driven by responses provided by users.As such, document sessions of the same multimodal application may vary.There is therefore an ongoing need for replay of document sessions inmultimodal applications.

SUMMARY OF THE INVENTION

Methods, apparatus, and computer program products are described fordocument session replay for multimodal applications. includingidentifying, by a multimodal browser in dependence upon a log producedby a Form Interpretation Algorithm (‘FIA’) during a previous documentsession with a user, a speech prompt provided by a multimodalapplication in the previous document session; identifying, by amultimodal browser in replay mode in dependence upon the log, a responseto the prompt provided by a user of the multimodal application in theprevious document session; retrieving, by the multimodal browser independence upon the log, an X+V page of the multimodal applicationassociated with the speech prompt and the response; rendering, by themultimodal browser, the visual elements of the retrieved X+V page;replaying, by the multimodal browser, the speech prompt; and replaying,by a multimodal browser, the response.

The foregoing and other objects, features and advantages of theinvention will be apparent from the following more particulardescriptions of exemplary embodiments of the invention as illustrated inthe accompanying drawings wherein like reference numbers generallyrepresent like parts of exemplary embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 sets forth a network diagram illustrating an exemplary system fordocument session replay for multimodal applications according toembodiments of the present invention

FIG. 2 sets forth a block diagram of automated computing machinerycomprising an example of a computer useful as a voice server in documentsession replay for multimodal applications according to embodiments ofthe present invention.

FIG. 3 sets forth a functional block diagram of exemplary apparatus fordocument session replay for multimodal applications in a thin clientarchitecture according to embodiments of the present invention.

FIG. 4 sets forth a block diagram of automated computing machinerycomprising an example of a computer useful as a multimodal device indocument session replay for multimodal applications according toembodiments of the present invention.

FIG. 5 sets forth a flow chart illustrating an exemplary method fordocument session replay for multimodal applications.

FIG. 6 sets forth additional aspects of some embodiments of documentsession replay for multimodal applications according to the presentinvention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Exemplary methods, apparatus, and products for document session replayfor multimodal applications are described with reference to theaccompanying drawings, beginning with FIG. 1. FIG. 1 sets forth anetwork diagram illustrating an exemplary system for document sessionreplay for multimodal applications according to embodiments of thepresent invention. Document session replay for multimodal applicationsin this example is implemented with a multimodal browser (196) and amultimodal application (195) operating on a multimodal device (152). Thesystem of FIG. 1 includes at least one speech recognition grammar (104)that specifies words and phrases to be recognized by an automatic speechrecognition (‘ASR’) engine (150) of a speech engine (148, 153). Themultimodal device (152) supports multiple modes of user interaction withthe multimodal application including a voice mode and one or morenon-voice modes of user interaction with the multimodal application. Thevoice mode is represented here with audio output of voice prompts andresponses (177) from the multimodal devices and audio input of speechfor recognition (315) from a user (128). Non-voice modes are representedby input/output devices such as keyboards and display screens on themultimodal devices (152). The multimodal application is operativelycoupled (195) to an ASR engine (150) in a speech engine (148). Theoperative coupling may be implemented with an application programminginterface (‘API’), a voice service module, or a VOIP connection asexplained more detail below.

The system of FIG. 1 operates generally to carry out document sessionreplay for multimodal applications according to embodiments of thepresent invention identifying, by a multimodal browser in dependenceupon a log produced by a Form Interpretation Algorithm (‘FIA’) during aprevious document session with a user, a speech prompt provided by amultimodal application in the previous document session; identifying, bya multimodal browser in replay mode in dependence upon the log, aresponse to the prompt provided by a user of the multimodal applicationin the previous document session; retrieving, by the multimodal browserin dependence upon the log, an X+V page of the multimodal applicationassociated with the speech prompt and the response; rendering, by themultimodal browser, the visual elements of the retrieved X+V page;replaying, by the multimodal browser, the speech prompt; and replaying,by a multimodal browser, the response

A document session is an implementation of the execution of a multimodalapplication including the prompts provided by the multimodal applicationand the responses received from a user of the multimodal application.Because multimodal applications are often driven by responses providedby users document sessions of the same multimodal application may vary.As such, document session replay provides a vehicle for reviewing thespecifics of a particular implementation of the multimodal applicationand specific responses in that implementation provided by a user.

A multimodal device is an automated device, that is, automated computingmachinery or a computer program running on an automated device, that iscapable of accepting from users more than one mode of input, keyboard,mouse, stylus, and so on, including speech input—and also displayingmore than one mode of output, graphic, speech, and so on. A multimodaldevice is generally capable of accepting speech input from a user,digitizing the speech, and providing digitized speech to a speech enginefor recognition. A multimodal device may be implemented, for example, asa voice-enabled browser on a laptop, a voice browser on a telephonehandset, an online game implemented with Java on a personal computer,and with other combinations of hardware and software as may occur tothose of skill in the art. Because multimodal applications may beimplemented in markup languages (X+V, SALT), object-oriented languages(Java, C++), procedural languages (the C programming language), and inother kinds of computer languages as may occur to those of skill in theart, this specification uses the term ‘multimodal application’ to referto any software application, server-oriented or client-oriented, thinclient or thick client, that administers more than one mode of input andmore than one mode of output, typically including visual and speechmodes.

The system of FIG. 1 includes several example multimodal devices:

-   -   personal computer (107) which is coupled for data communications        to data communications network (100) through wireline connection        (120),    -   personal digital assistant (‘PDA’) (112) which is coupled for        data communications to data communications network (100) through        wireless connection (114),    -   mobile telephone (110) which is coupled for data communications        to data communications network (100) through wireless connection        (116), and    -   laptop computer (126) which is coupled for data communications        to data communications network (100) through wireless connection        (118).

Each of the example multimodal devices (152) in the system of FIG. 1includes a microphone, an audio amplifier, a digital-to-analogconverter, and a multimodal application capable of accepting from a user(128) speech for recognition (315), digitizing the speech, and providingthe digitized speech to a speech engine for recognition. The speech maybe digitized according to industry standard codecs, including but notlimited to those used for Distributed Speech Recognition as such.Methods for ‘COding/DECoding’ speech are referred to as ‘codecs.’ TheEuropean Telecommunications Standards Institute (‘ETSI’) providesseveral codecs for encoding speech for use in DSR, including, forexample, the ETSI ES 201 108 DSR Front-end Codec, the ETSI ES 202 050Advanced DSR Front-end Codec, the ETSI ES 202 211 Extended DSR Front-endCodec, and the ETSI ES 202 212 Extended Advanced DSR Front-end Codec. Instandards such as RFC3557 entitled

-   -   RTP Payload Format for European Telecommunications Standards        Institute (ETSI) European Standard ES 201 108 Distributed Speech        Recognition Encoding        and the Internet Draft entitled    -   RTP Payload Formats for European Telecommunications Standards        Institute (ETSI) European Standard ES 202 050, ES 202 211, and        ES 202 212 Distributed Speech Recognition Encoding,        the IETF provides standard RTP payload formats for various        codecs. It is useful to note, therefore, that there is no        limitation in the present invention regarding codecs, payload        formats, or packet structures. Speech for document session        replay according to embodiments of the present invention may be        encoded with any codec, including, for example:    -   AMR (Adaptive Multi-Rate Speech coder)    -   ARDOR (Adaptive Rate-Distortion Optimized sound codeR),    -   Dolby Digital (A/52, AC3),    -   DTS (DTS Coherent Acoustics),    -   MP1 (MPEG audio layer-1),    -   MP2 (MPEG audio layer-2) Layer 2 audio codec (MPEG-1, MPEG-2 and        non-ISO MPEG-2.5),    -   MP3 (MPEG audio layer-3) Layer 3 audio codec (MPEG-1, MPEG-2 and        non-ISO MPEG-2.5),    -   Perceptual Audio Coding,    -   FS-1015 (LPC-10),    -   FS-1016 (CELP),    -   G.726 (ADPCM),    -   G.728 (LD-CELP),    -   G.729 (CS-ACELP),    -   GSM,    -   HILN (MPEG-4 Parametric audio coding), and    -   others as may occur to those of skill in the art.

As mentioned, a multimodal device according to embodiments of thepresent invention is capable of providing speech to a speech engine forrecognition. A speech engine is a functional module, typically asoftware module, although it may include specialized hardware also, thatdoes the work of recognizing and generating or ‘synthesizing’ humanspeech. The speech engine implements speech recognition by use of afurther module referred to in this specification as a ASR engine, andthe speech engine carries out speech synthesis by use of a furthermodule referred to in this specification as a text-to-speech (‘TTS’)engine. As shown in FIG. 1, a speech engine (148) may be installedlocally in the multimodal device (107) itself, or a speech engine (153)may be installed remotely with respect to the multimodal device, acrossa data communications network (100) in a voice server (151). Amultimodal device that itself contains its own speech engine is said toimplement a ‘thick multimodal client’ or ‘thick client,’ because thethick multimodal client device itself contains all the functionalityneeded to carry out speech recognition and speech synthesis—through APIcalls to speech recognition and speech synthesis modules in themultimodal device itself with no need to send requests for speechrecognition across a network and no need to receive synthesized speechacross a network from a remote voice server. A multimodal device thatdoes not contain its own speech engine is said to implement a ‘thinmultimodal client’ or simply a ‘thin client,’ because the thinmultimodal client itself contains only a relatively thin layer ofmultimodal application software that obtains speech recognition andspeech synthesis services from a voice server located remotely across anetwork from the thin client. For ease of explanation, only one (107) ofthe multimodal devices (152) in the system of FIG. 1 is shown with aspeech engine (148), but readers will recognize that any multimodaldevice may have a speech engine according to embodiments of the presentinvention.

A multimodal application (195) in this example provides speech forrecognition and text for speech synthesis to a speech engine through aVoiceXML interpreter (149, 155). A VoiceXML interpreter is a softwaremodule of computer program instructions that accepts voice dialoginstructions from a multimodal application, typically in the form of aVoiceXML <form> element. The voice dialog instructions include one ormore grammars, data input elements, event handlers, and so on, thatadvise the VoiceXML interpreter how to administer voice input from auser and voice prompts and responses to be presented to a user. TheVoiceXML interpreter administers such dialogs by processing the dialoginstructions sequentially in accordance with a VoiceXML FormInterpretation Algorithm (‘FIA’).

A Form Interpretation Algorithm (‘FIA’) drives the interaction betweenthe user and a multimodal application. The FIA is generally responsiblefor selecting and playing one or more speech prompts, collecting a userinput, either a response that fills in one or more input items, or athrowing of some event, and interpreting actions that pertained to thenewly filled in input items. The FIA also handles multimodal applicationinitialization, management of the prompt counters needed for prompttapering, grammar activation and deactivation, entering and leavingforms with matching utterances and many other tasks.

The FIA maintains a detailed log of actions of a running multimodalapplication useful in document session replay according to the presentinvention. The log includes stamped actions carried out in a documentsession of a multimodal application including speech prompts provided toa user, identifications of X+V pages of the multimodal applicationpresented to a user, responses received from a user, current state frameinformation such as the values of Voice XML shadow variables, values ofDOM elements, threads of execution, as well as other information as willoccur to those of skill in the art.

As shown in FIG. 1, a VoiceXML interpreter (149) may be installedlocally in the multimodal device (107) itself, or a VoiceXML interpreter(155) may be installed remotely with respect to the multimodal device,across a data communications network (100) in a voice server (151). In athick client architecture, a multimodal device (152) includes both itsown speech engine (148) and its own VoiceXML interpreter (149). TheVoiceXML interpreter (149) exposes an API to the multimodal application(195) for use in providing speech recognition and speech synthesis forthe multimodal application. The multimodal application provides dialoginstructions, VoiceXML <form> elements, grammars, input elements, eventhandlers, and so on, through the API to the VoiceXML interpreter, andthe VoiceXML interpreter administers the speech engine on behalf of themultimodal application. In the thick client architecture, VoiceXMLdialogs are interpreted by a VoiceXML interpreter on the multimodaldevice. In the thin client architecture, VoiceXML dialogs areinterpreted by a VoiceXML interpreter on a voice server (151) locatedremotely across a data communications network (100) from the multimodaldevice running the multimodal application (195).

The VoiceXML interpreter provides grammars, speech for recognition, andtext prompts for speech synthesis to the speech engine, and the VoiceXMLinterpreter returns to the multimodal application speech engine outputin the form of recognized speech, semantic interpretation results, anddigitized speech for voice prompts. In a thin client architecture, theVoiceXML interpreter (155) is located remotely from the multimodalclient device in a voice server (151), the API for the VoiceXMLinterpreter is still implemented in the multimodal device, with the APImodified to communicate voice dialog instructions, speech forrecognition, and text and voice prompts to and from the VoiceXMLinterpreter on the voice server. For ease of explanation, only one (107)of the multimodal devices (152) in the system of FIG. 1 is shown with aVoiceXML interpreter (149), but readers will recognize that anymultimodal device may have a VoiceXML interpreter according toembodiments of the present invention. Each of the example multimodaldevices (152) in the system of FIG. 1 may be configured to carry outdocument session replay for multimodal applications installing andrunning on the multimodal device a multimodal browser that carries outdocument session replay for multimodal applications according toembodiments of the present invention.

The use of these four example multimodal devices (152) is forexplanation only, not for limitation of the invention. Any automatedcomputing machinery capable of accepting speech from a user, providingthe speech digitized to an ASR engine through a VoiceXML interpreter,and receiving and playing speech prompts and responses from the VoiceXMLinterpreter may be improved to function as a multimodal device fordocument session replay for multimodal applications according toembodiments of the present invention.

The system of FIG. 1 also includes a voice server (151) which isconnected to data communications network (100) through wirelineconnection (122). The voice server (151) is a computer that runs aspeech engine (153) that provides voice recognition services formultimodal devices by accepting requests for speech recognition andreturning text representing recognized speech. Voice server (151) alsoprovides speech synthesis, text to speech (‘TTS’) conversion, for voiceprompts and voice responses (314) to user input in multimodalapplications such as, for example, X+V applications, SALT applications,or Java voice applications.

The system of FIG. 1 includes a data communications network (100) thatconnects the multimodal devices (152) and the voice server (151) fordata communications. A data communications network for document sessionreplay for multimodal applications according to embodiments of thepresent invention is a data communications data communications networkcomposed of a plurality of computers that function as datacommunications routers connected for data communications with packetswitching protocols. Such a data communications network may beimplemented with optical connections, wireline connections, or withwireless connections. Such a data communications network may includeintranets, internets, local area data communications networks (‘LANs’),and wide area data communications networks (‘WANs’). Such a datacommunications network may implement, for example:

-   -   a link layer with the Ethernet™ Protocol or the Wireless        Ethernet™ Protocol,    -   a data communications network layer with the Internet Protocol        (‘IP’),    -   a transport layer with the Transmission Control Protocol (‘TCP’)        or the User Datagram Protocol (‘UDP’),    -   an application layer with the HyperText Transfer Protocol        (‘HTTP’), the Session Initiation Protocol (‘SIP’), the Real Time        Protocol (‘RTP’), the Distributed Multimodal Synchronization        Protocol (‘DMSP’), the Wireless Access Protocol (‘WAP’), the        Handheld Device Transfer Protocol (‘HDTP’), the ITU protocol        known as H.323, and    -   other protocols as will occur to those of skill in the art.

The system of FIG. 1 includes a web server (147) connected for datacommunications through wireline connection (123) to network (100) andtherefore to the multimodal devices (152). The web server (147) may beany server that provides to client devices markup documents that composemultimodal applications. The web server (147) typically provides suchmarkup documents via a data communications protocol, HTTP, HDTP, WAP, orthe like. That is, although the term ‘web’ is used to described the webserver generally in this specification, there is no limitation of datacommunications between multimodal devices and the web server to HTTPalone. The markup documents also may be implemented in any markuplanguage that supports non-speech display elements, data entry elements,and speech elements for identifying which speech to recognize and whichwords to speak, grammars, form elements, and the like, including, forexample, X+V and SALT. A multimodal application in a multimodal devicethen, upon receiving from the web sever (147) a markup document as partof a multimodal application, may execute speech elements by use of aVoiceXML interpreter (149) and speech engine (148) in the multimodaldevice itself or by use of a VoiceXML interpreter (155) and speechengine (153) located remotely from the multimodal device in a voiceserver (151).

The arrangement of the multimodal devices (152), the web server (147),the voice server (151), and the data communications network (100) makingup the exemplary system illustrated in FIG. 1 are for explanation, notfor limitation. Data processing systems useful for document sessionreplay for multimodal applications speech recognition according tovarious embodiments of the present invention may include additionalservers, routers, other devices, and peer-to-peer architectures, notshown in FIG. 1, as will occur to those of skill in the art. Datacommunications networks in such data processing systems may support manydata communications protocols in addition to those noted above. Variousembodiments of the present invention may be implemented on a variety ofhardware platforms in addition to those illustrated in FIG. 1.

Document session replay for multimodal applications according toembodiments of the present invention in a thin client architecture maybe implemented with one or more voice servers, computers, that is,automated computing machinery, that provide speech recognition andspeech synthesis. For further explanation, therefore, FIG. 2 sets fortha block diagram of automated computing machinery comprising an exampleof a computer useful as a voice server (151) in document session replayfor multimodal applications according to embodiments of the presentinvention. The voice server (151) of FIG. 2 includes at least onecomputer processor (156) or ‘CPU’ as well as random access memory (168)(‘RAM’) which is connected through a high speed memory bus (166) and busadapter (158) to processor (156) and to other components of the voiceserver.

Stored in RAM (168) is a voice server application (188), a module ofcomputer program instructions capable of operating a voice server in asystem that is configured to carry out document session replay formultimodal applications according to embodiments of the presentinvention. Voice server application (188) provides voice recognitionservices for multimodal devices by accepting requests for speechrecognition and returning speech recognition results, including textrepresenting recognized speech, text for use as variable values indialogs, and text as string representations of scripts for semanticinterpretation. Voice server application (188) also includes computerprogram instructions that provide text-to-speech (‘TTS’) conversion forvoice prompts and voice responses to user input in multimodalapplications such as, for example, X+V applications, SALT applications,or Java Speech applications.

Voice server application (188) may be implemented as a web server,implemented in Java, C++, or another language, that supports X+V, SALT,VoiceXML, or other multimodal languages, by providing responses to HTTPrequests from X+V clients, SALT clients, Java Speech clients, or othermultimodal clients. Voice server application (188) may, for a furtherexample, be implemented as a Java server that runs on a Java VirtualMachine (102) and supports a Java voice framework by providing responsesto HTTP requests from Java client applications running on multimodaldevices. And voice server applications that support embodiments of thepresent invention may be implemented in other ways as may occur to thoseof skill in the art, and all such ways are well within the scope of thepresent invention.

The voice server (151) in this example includes a speech engine (153).The speech engine is a functional module, typically a software module,although it may include specialized hardware also, that does the work ofrecognizing and generating human speech. The speech engine (153)includes an automated speech recognition (‘ASR’) engine for speechrecognition and a text-to-speech (‘TTS’) engine for generating speech.The speech engine also includes a grammar (104), a lexicon (106), and alanguage-specific acoustic model (108). The language-specific acousticmodel (108) is a data structure, a table or database, for example, thatassociates SFVs with phonemes representing, to the extent that it ispractically feasible to do so, all pronunciations of all the words in ahuman language. The lexicon (106) is an association of words in textform with phonemes representing pronunciations of each word; the lexiconeffectively identifies words that are capable of recognition by an ASRengine. Also stored in RAM (168) is a Text To Speech (‘TTS’) Engine(194), a module of computer program instructions that accepts text asinput and returns the same text in the form of digitally encoded speech,for use in providing speech as prompts for and responses to users ofmultimodal systems.

The grammar (104) communicates to the ASR engine (150) the words andsequences of words that currently may be recognized. For preciseunderstanding, distinguish the purpose of the grammar and the purpose ofthe lexicon. The lexicon associates with phonemes all the words that theASR engine can recognize. The grammar communicates the words currentlyeligible for recognition. The set of words currently eligible forrecognition and the set of words capable of recognition may or may notbe the same.

Grammars for use in document session replay for multimodal applicationsaccording to embodiments of the present invention may be expressed inany format supported by any ASR engine, including, for example, the JavaSpeech Grammar Format (‘JSGF’), the format of the W3C Speech RecognitionGrammar Specification (‘SRGS’), the Augmented Backus-Naur Format(‘ABNF’) from the IETF's RFC2234, in the form of a stochastic grammar asdescribed in the W3C's Stochastic Language Models (N-Gram)Specification, and in other grammar formats as may occur to those ofskill in the art. Grammars typically operate as elements of dialogs,such as, for example, a VoiceXML <menu> or an X+V <form>. A grammar'sdefinition may be expressed in-line in a dialog. Or the grammar may beimplemented externally in a separate grammar document and referencedfrom with a dialog with a URI. Here is an example of a grammar expressedin JSFG:

<grammar scope=“dialog” ><![CDATA[   #JSGF V1.0;   grammar command;  <command> = [remind me to] call | phone |   telephone <name> <when>;  <name> = bob | martha | joe | pete | chris | john | artoush;   <when>= today | this afternoon | tomorrow | next week;   ]]> </grammar>

In this example, the elements named <command>, <name>, and <when> arerules of the grammar. Rules are a combination of a rulename and anexpansion of a rule that advises an ASR engine or a voice interpreterwhich words presently can be recognized. In this example, expansionincludes conjunction and disjunction, and the vertical bars ‘|’ mean‘or.’ An ASR engine or a voice interpreter processes the rules insequence, first <command>, then <name>, then <when>. The <command> ruleaccepts for recognition ‘call’ or ‘phone’ or ‘telephone’ plus, that is,in conjunction with, whatever is returned from the <name> rule and the<when> rule. The <name> rule accepts ‘bob’ or ‘martha’ or ‘joe’ or‘pete’ or ‘chris’ or ‘john’ or ‘artoush’, and the <when> rule accepts‘today’ or ‘this afternoon’ or ‘tomorrow’ or ‘next week.’ The commandgrammar as a whole matches utterances like these, for example:

-   -   “phone bob next week,”    -   “telephone martha this afternoon,”    -   “remind me to call chris tomorrow,” and    -   “remind me to phone pete today.”

The voice server application (188) in this example is configured toreceive, from a multimodal client located remotely across a network fromthe voice server, digitized speech for recognition from a user and passthe speech along to the ASR engine (150) for recognition. ASR engine(150) is a module of computer program instructions, also stored in RAMin this example. In carrying out automated speech recognition, the ASRengine receives speech for recognition in the form of at least onedigitized word and uses frequency components of the digitized word toderive a Speech Feature Vector (‘SFV’). An SFV may be defined, forexample, by the first twelve or thirteen Fourier or frequency domaincomponents of a sample of digitized speech. The ASR engine can use theSFV to infer phonemes for the word from the language-specific acousticmodel (108). The ASR engine then uses the phonemes to find the word inthe lexicon (106).

Also stored in RAM is a VoiceXML interpreter (192), a module of computerprogram instructions that processes VoiceXML grammars. VoiceXML input toVoiceXML interpreter (192) may originate, for example, from VoiceXMLclients running remotely on multimodal devices, from X+V clients runningremotely on multimodal devices, from SALT clients running on multimodaldevices, or from Java client applications running remotely on multimediadevices. In this example, VoiceXML interpreter (192) interprets andexecutes VoiceXML segments representing voice dialog instructionsreceived from remote multimedia devices and provided to VoiceXMLinterpreter (192) through voice server application (188).

A multimodal application (195) in a thin client architecture may providevoice dialog instructions, VoiceXML segments, VoiceXML <form> elements,and the like, to VoiceXML interpreter (149) through data communicationsacross a network with multimodal application (195). The voice dialoginstructions include one or more grammars, data input elements, eventhandlers, and so on, that advise the VoiceXML interpreter how toadminister voice input from a user and voice prompts and responses to bepresented to a user. The VoiceXML interpreter administers such dialogsby processing the dialog instructions sequentially in accordance with aVoiceXML Form Interpretation Algorithm (‘FIA’) (193). The VoiceXMLinterpreter interprets VoiceXML dialogs provided to the VoiceXMLinterpreter by a multimodal application.

As mentioned above, a Form Interpretation Algorithm (‘FIA’) drives theinteraction between the user and a multimodal application. The FIA isgenerally responsible for selecting and playing one or more speechprompts, collecting a user input, either a response that fills in one ormore input items, or a throwing of some event, and interpreting actionsthat pertained to the newly filled in input items. The FIA also handlesmultimodal application initialization, management of the prompt countersneeded for prompt tapering, grammar activation and deactivation,entering and leaving forms with matching utterances and many othertasks.

The FIA (193) of FIG. 2 maintains a detailed log (504) of actions of arunning multimodal application useful in document session replayaccording to the present invention. The log includes stamped actionscarried out in a document session of a multimodal application includingspeech prompts provided to a user, identifications of X+V pages of themultimodal application presented to a user, responses received from auser, current state frame information such as the values of Voice XMLshadow variables, values of DOM elements, threads of execution, as wellas other information as will occur to those of skill in the art.

Also stored in RAM (168) is an operating system (154). Operating systemsuseful in voice servers according to embodiments of the presentinvention include UNIX™, Linux™, Microsoft NT™, AIX™, IBM's i5/OS™, andothers as will occur to those of skill in the art. Operating system(154), voice server application (188), VoiceXML interpreter (192), ASRengine (150), JVM (102), and TTS Engine (194) in the example of FIG. 2are shown in RAM (168), but many components of such software typicallyare stored in non-volatile memory also, for example, on a disk drive(170).

Voice server (151) of FIG. 2 includes bus adapter (158), a computerhardware component that contains drive electronics for high speed buses,the front side bus (162), the video bus (164), and the memory bus (166),as well as drive electronics for the slower expansion bus (160).Examples of bus adapters useful in voice servers according toembodiments of the present invention include the Intel Northbridge, theIntel Memory Controller Hub, the Intel Southbridge, and the Intel I/OController Hub. Examples of expansion buses useful in voice serversaccording to embodiments of the present invention include IndustryStandard Architecture (‘ISA’) buses and Peripheral ComponentInterconnect (‘PCI’) buses.

Voice server (151) of FIG. 2 includes disk drive adapter (172) coupledthrough expansion bus (160) and bus adapter (158) to processor (156) andother components of the voice server (151). Disk drive adapter (172)connects non-volatile data storage to the voice server (151) in the formof disk drive (170). Disk drive adapters useful in voice servers includeIntegrated Drive Electronics (‘IDE’) adapters, Small Computer SystemInterface (‘SCSI’) adapters, and others as will occur to those of skillin the art. In addition, non-volatile computer memory may be implementedfor a voice server as an optical disk drive, electrically erasableprogrammable read-only memory (so-called ‘EEPROM’ or ‘Flash’ memory),RAM drives, and so on, as will occur to those of skill in the art.

The example voice server of FIG. 2 includes one or more input/output(‘I/O’) adapters (178). I/O adapters in voice servers implementuser-oriented input/output through, for example, software drivers andcomputer hardware for controlling output to display devices such ascomputer display screens, as well as user input from user input devices(181) such as keyboards and mice. The example voice server of FIG. 2includes a video adapter (209), which is an example of an I/O adapterspecially designed for graphic output to a display device (180) such asa display screen or computer monitor. Video adapter (209) is connectedto processor (156) through a high speed video bus (164), bus adapter(158), and the front side bus (162), which is also a high speed bus.

The exemplary voice server (151) of FIG. 2 includes a communicationsadapter (167) for data communications with other computers (182) and fordata communications with a data communications network (100). Such datacommunications may be carried out serially through RS-232 connections,through external buses such as a Universal Serial Bus (‘USB’), throughdata communications data communications networks such as IP datacommunications networks, and in other ways as will occur to those ofskill in the art. Communications adapters implement the hardware levelof data communications through which one computer sends datacommunications to another computer, directly or through a datacommunications network. Examples of communications adapters useful forembodiments of the present invention include modems for wired dial-upcommunications, Ethernet (IEEE 802.3) adapters for wired datacommunications network communications, and 802.11 adapters for wirelessdata communications network communications.

For further explanation, FIG. 3 sets forth a functional block diagram ofexemplary apparatus for document session replay for multimodalapplications in a thin client architecture according to embodiments ofthe present invention. The example of FIG. 3 includes a multimodaldevice (152) and a voice server (151) connected for data communicationby a VOIP connection (216) through a data communications network (100).A multimodal application (195) runs on the multimodal device (152), anda voice server application (188) runs on the voice server (151). Themultimodal client application (195) may be a set or sequence of X+V orSALT documents that execute on multimodal browser (196), a Java voiceapplication that executes on the Java Virtual Machine (101), or amultimodal application implemented in other technologies as may occur tothose of skill in the art. The example multimodal device of FIG. 3 alsoincludes a sound card (174), which is an example of an I/O adapterspecially designed for accepting analog audio signals from a microphone(176) and converting the audio analog signals to digital form forfurther processing by a codec (183).

In addition to the multimodal sever application (188), the voice server(151) also has installed upon it a speech engine (153) with an ASRengine (150), a grammar (104), a lexicon (106), a language-specificacoustic model (108), and a TTS engine (194), as well as a JVM (102),and a Voice XML interpreter (192). VoiceXML interpreter (192) interpretsand executes VoiceXML dialog instructions received from the multimodalapplication and provided to VoiceXML interpreter (192) through voiceserver application (188). VoiceXML input to VoiceXML interpreter (192)may originate from the multimodal application (195) implemented as anX+V client running remotely on the multimodal device (152). As notedabove, the multimedia device application (195) also may be implementedas a Java client application running remotely on the multimedia device(152), a SALT application running remotely on the multimedia device(152), and in other ways as may occur to those of skill in the art.

The FIA (193) of FIG. 3 maintains a detailed log of actions of a runningmultimodal application useful in document session replay according tothe present invention. The log includes stamped actions carried out in adocument session of a multimodal application including speech promptsprovided to a user, identifications of X+V pages of the multimodalapplication presented to a user, responses received from a user, currentstate frame information such as the values of Voice XML shadowvariables, values of DOM elements, threads of execution, as well asother information as will occur to those of skill in the art.

VOIP stands for ‘Voice Over Internet Protocol,’ a generic term forrouting speech over an IP-based data communications network. The speechdata flows over a general-purpose packet-switched data communicationsnetwork, instead of traditional dedicated, circuit-switched voicetransmission lines. Protocols used to carry voice signals over the IPdata communications network are commonly referred to as ‘Voice over IP’or ‘VOIP’ protocols. VOIP traffic may be deployed on any IP datacommunications network, including data communications networks lacking aconnection to the rest of the Internet, for instance on a privatebuilding-wide local area data communications network or ‘LAN.’

Many protocols are used to effect VOIP. The two most popular types ofVOIP are effected with the IETF's Session Initiation Protocol (‘SIP’)and the ITU's protocol known as ‘H.323.’ SIP clients use TCP and UDPport 5060 to connect to SIP servers. SIP itself is used to set up andtear down calls for speech transmission. VOIP with SIP then uses RTP fortransmitting the actual encoded speech. Similarly, H.323 is an umbrellarecommendation from the standards branch of the InternationalTelecommunications Union that defines protocols to provide audio-visualcommunication sessions on any packet data communications network.

The apparatus of FIG. 3 operates in a manner that is similar to theoperation of the system of FIG. 2 described above. Multimodalapplication (195) is a user-level, multimodal, client-side computerprogram that presents a voice interface to user (128), provides audioprompts and responses (314) and accepts input speech for recognition(315). Multimodal application (195) provides a speech interface throughwhich a user may provide oral speech for recognition through microphone(176) and have the speech digitized through an audio amplifier (185) anda coder/decoder (‘codec’) (183) of a sound card (174) and provide thedigitized speech for recognition to ASR engine (150). Multimodalapplication (195) then packages the digitized speech in a recognitionrequest message according to a VOIP protocol, and transmits the speechto voice server (151) through the VOIP connection (216) on the network(100).

Voice server application (188) provides voice recognition services formultimodal devices by accepting dialog instructions, VoiceXML segments,and returning speech recognition results, including text representingrecognized speech, text for use as variable values in dialogs, andoutput from execution of semantic interpretation scripts as well asvoice prompts. Voice server application (188) includes computer programinstructions that provide text-to-speech (‘TTS’) conversion for voiceprompts and voice responses to user input in multimodal applicationssuch as, for example, X+V applications, SALT applications, or JavaSpeech applications.

The voice server application (188) receives speech for recognition froma user and passes the speech through API calls to VoiceXML interpreter(192) which in turn uses an ASR engine (150) for speech recognition. TheASR engine receives digitized speech for recognition, uses frequencycomponents of the digitized speech to derive an SFV, uses the SFV toinfer phonemes for the word from the language-specific acoustic model(108), and uses the phonemes to find the speech in the lexicon (106).The ASR engine then compares speech found as words in the lexicon towords in a grammar (104) to determine whether words or phrases in speechare recognized by the ASR engine.

The multimodal application (195) is operatively coupled to the ASRengine (150). In this example, the operative coupling between themultimodal application and the ASR engine (150) is implemented with aVOIP connection (216) through a voice services module (130), thenthrough the voice server application (188) and either JVM (102),VoiceXML interpreter (192), or SALT interpreter (103), depending onwhether the multimodal application is implemented in X+V, Java, or SALT.The voice services module (130) is a thin layer of functionality, amodule of computer program instructions, that presents an API (316) foruse by an application level program in providing dialog instructions andspeech for recognition to a voice server application (188) and receivingin response voice prompts and other responses. In this example,application level programs are represented by multimodal application(195), JVM (101), and multimodal browser (196).

The voice services module (130) provides data communications servicesthrough the VOIP connection and the voice server application (188)between the multimodal device (152) and the VoiceXML interpreter (192).The API (316) is the same API presented to applications by a VoiceXMLinterpreter when the VoiceXML interpreter is installed on the multimodaldevice in a thick client architecture (316 on FIG. 4). So from the pointof view of an application calling the API (316), the application iscalling the VoiceXML interpreter directly. The data communicationsfunctions of the voice services module (130) are transparent toapplications that call the API (316). At the application level, calls tothe API (316) may be issued from the multimodal browser (196), whichprovides an execution environment for the multimodal application (195)when the multimodal application is implemented with X+V. And calls tothe API (316) may be issued from the JVM (101), which provides anexecution environment for the multimodal application (195) when themultimodal application is implemented with Java.

The multimodal browser (196) of FIG. 3 includes a record module (350)computer program instructions for recording the speech responsesprovided by a ser (128). Such speech responses are recorded for use indocument session replay according to embodiments of the presentinvention.

The multimodal browser (196) of FIG. 3 also includes a replay module(350) computer program instructions for document session replay formultimodal applications. The replay module (350) of FIG. 3 operatesgenerally by identifying, in dependence upon a log (504) produced by aForm Interpretation Algorithm (‘FIA’) (193) during a previous documentsession with a user (128), a speech prompt provided by a multimodalapplication (195) in the previous document session; identifying, independence upon the log (504), a response to the prompt provided by auser (128) of the multimodal application (195) in the previous documentsession; retrieving, in dependence upon the log (504), an X+V page (522)of the multimodal application (195) associated with the speech promptand the response; rendering the visual elements of the retrieved X+Vpage (522).

The replay module (350) of FIG. 3 also operates generally by replayingthe speech prompt. Replaying the speech prompt may be carried out byreplaying from the log (504) the speech prompt from the previousdocument session. Replaying the speech prompt may be carried out byreplaying with the multimodal browser (196) the speech prompt from theretrieved page (522) of the previous document session.

The replay module (350) of FIG. 3 also operates generally by replayingthe response. The response may be either a voice mode response or anon-voice mode response. Where the response is a voice mode responsefrom the user, replaying the response includes retrieving a recording ofthe voice mode response and playing the recording. Where the response isa non-voice mode response, replaying the response includes recreatingthe non-voice mode response.

Document session replay for multimodal applications according toembodiments of the present invention in thick client architectures isgenerally implemented with multimodal devices, that is, automatedcomputing machinery or computers. In the system of FIG. 1, for example,all the multimodal devices (152) are implemented to some extent at leastas computers. For further explanation, therefore, FIG. 4 sets forth ablock diagram of automated computing machinery comprising an example ofa computer useful as a multimodal device (152) in document sessionreplay for multimodal applications according to embodiments of thepresent invention. In a multimodal device implementing a thick clientarchitecture as illustrated in FIG. 4, the multimodal device (152) hasno connection to a remote voice server containing a VoiceXML interpreterand a speech engine. All the components needed for speech synthesis andvoice recognition according to embodiments of the present invention areinstalled or embedded in the multimodal device itself.

The example multimodal device (152) of FIG. 4 includes severalcomponents that are structured and operate similarly as do parallelcomponents of the voice server, having the same drawing referencenumbers, as described above with reference to FIG. 2: at least onecomputer processor (156), frontside bus (162), RAM (168), high speedmemory bus (166), bus adapter (158), video adapter (209), video bus(164), expansion bus (160), communications adapter (167), I/O adapter(178), disk drive adapter (172), an operating system (154), a JVM (102),a VoiceXML Interpreter (192), a speech engine (153), and so on. As inthe system of FIG. 4, the speech engine in the multimodal device of FIG.2 includes an ASR engine (150), a grammar (104), a lexicon (106), alanguage-dependent acoustic model (108), and a TTS engine (194). TheVoiceXML interpreter (192) administers such dialogs by processing thedialog instructions sequentially in accordance with a VoiceXML FormInterpretation Algorithm (‘FIA’) (193). The FIA (193) of FIG. 3maintains a detailed log of actions of a running multimodal applicationuseful in document session replay according to the present invention.

The speech engine (153) in this kind of embodiment, a thick clientarchitecture, often is implemented as an embedded module in a small formfactor device such as a handheld device, a mobile phone, PDA, and thelike. An example of an embedded speech engine useful for embodiments ofthe present invention is IBM's Embedded ViaVoice Enterprise. The examplemultimodal device of FIG. 4 also includes a sound card (174), which isan example of an I/O adapter specially designed for accepting analogaudio signals from a microphone (176) and converting the audio analogsignals to digital form for further processing by a codec (183). Thesound card (174) is connected to processor (156) through expansion bus(160), bus adapter (158), and front side bus (162).

Also stored in RAM (168) in this example is a multimodal application(195), a module of computer program instructions capable of operating amultimodal device as an apparatus that supports embodiments of thepresent invention. The multimodal application (195) implements speechrecognition by accepting speech for recognition from a user and sendingthe speech for recognition through API calls to the ASR engine (150).The multimodal application (195) implements speech synthesis generallyby sending words to be used as prompts for a user to the TTS engine(194). As an example of thick client architecture, the multimodalapplication (195) in this example does not send speech for recognitionacross a network to a voice server for recognition, and the multimodalapplication (195) in this example does not receive synthesized speech,TTS prompts and responses, across a network from a voice server. Allgrammar processing, voice recognition, and text to speech conversion inthis example is performed in an embedded fashion in the multimodaldevice (152) itself.

More particularly, multimodal application (195) in this example is auser-level, multimodal, client-side computer program that provides aspeech interface through which a user may provide oral speech forrecognition through microphone (176), have the speech digitized throughan audio amplifier (185) and a coder/decoder (‘codec’) (183) of a soundcard (174) and provide the digitized speech for recognition to ASRengine (150). The multimodal application (195) may be implemented as aset or sequence of X+V documents executing in a multimodal browser (196)or microbrowser that passes VoiceXML grammars and digitized speech bycalls through an API (316) directly to an embedded VoiceXML interpreter(192) for processing. The embedded VoiceXML interpreter (192) may inturn issue requests for speech recognition through API calls directly tothe embedded ASR engine (150). Multimodal application (195) also canprovide speech synthesis, TTS conversion, by API calls to the embeddedTTS engine (194) for voice prompts and voice responses to user input.

In a further class of exemplary embodiments, the multimodal application(195) may be implemented as a Java voice application that executes onJava Virtual Machine (102) and issues calls through the VoiceXML API(316) for speech recognition and speech synthesis services. In furtherexemplary embodiments, the multimodal application (195) may beimplemented as a set or sequence of SALT documents executed on amultimodal browser (196) or microbrowser that issues calls through theVoiceXML API (316) for speech recognition and speech synthesis services.In addition to X+V, SALT, and Java implementations, multimodalapplication (195) may be implemented in other technologies as will occurto those of skill in the art, and all such implementations are wellwithin the scope of the present invention.

The multimodal application (195) is operatively coupled to the ASRengine (150). In this example, the operative coupling between themultimodal application and the ASR engine (150) is implemented eitherJVM (102), VoiceXML interpreter (192), or SALT interpreter (103),depending on whether the multimodal application is implemented in X+V,Java, or SALT. When the multimodal application (195) is implemented inX+V, the operative coupling is effected through the multimodal browser(196), which provides an operating environment and an interpreter forthe X+V application, and then through the VoiceXML interpreter, whichpasses grammars and voice utterances for recognition to the ASR engine.When the multimodal application (195) is implemented in Java Speech, theoperative coupling is effected through the JVM (102), which provides anoperating environment for the Java application and passes grammars andvoice utterances for recognition to the ASR engine. When the multimodalapplication (195) is implemented in SALT, the operative coupling iseffected through the SALT interpreter (103), which provides an operatingenvironment and an interpreter for the X+V application and passesgrammars and voice utterances for recognition to the ASR engine.

The multimodal application (195) in this example, running on amultimodal device (152) that contains its own VoiceXML interpreter (192)and its own speech engine (153) with no network or VOIP connection to aremote voice server containing a remote VoiceXML interpreter or a remotespeech engine, is an example of a so-called ‘thick client architecture,’so-called because all of the functionality for processing voice modeinteractions between a user and the multimodal application isimplemented on the multimodal device itself.

The multimodal browser (196) of FIG. 4 includes a record module (350)computer program instructions for recording the speech responsesprovided by a ser (128). Such speech responses are recorded for use indocument session replay according to embodiments of the presentinvention.

The multimodal browser (196) of FIG. 4 also includes a replay module(350) computer program instructions for document session replay formultimodal applications. The replay module (350) of FIG. 4 operatesgenerally by identifying, in dependence upon a log (504) produced by aForm Interpretation Algorithm (‘FIA’) (193) during a previous documentsession with a user (128), a speech prompt provided by a multimodalapplication (195) in the previous document session; identifying, independence upon the log (504), a response to the prompt provided by auser (128) of the multimodal application (195) in the previous documentsession; retrieving, in dependence upon the log (504), an X+V page (522)of the multimodal application (195) associated with the speech promptand the response; rendering the visual elements of the retrieved X+Vpage (522).

The replay module (350) of FIG. 4 also operates generally by replayingthe speech prompt. Replaying the speech prompt may be carried out byreplaying from the log (504) the speech prompt from the previousdocument session. Replaying the speech prompt may be carried out byreplaying with the multimodal browser (196) the speech prompt from theretrieved page (522) of the previous document session.

The replay module (350) of FIG. 4 also operates generally by replayingthe response. The response may be either a voice mode response or anon-voice mode response. Where the response is a voice mode responsefrom the user, replaying the response includes retrieving a recording ofthe voice mode response and playing the recording. Where the response isa non-voice mode response, replaying the response includes recreatingthe non-voice mode response.

For further explanation, FIG. 5 sets forth a flow chart illustrating anexemplary method for document session replay for multimodalapplications. As mentioned above, a document session is animplementation of the execution of a multimodal application includingthe prompts provided by the multimodal application and the responsesreceived from a user of the multimodal application. Because multimodalapplications are often driven by responses provided by users documentsessions of the same multimodal application may vary. As such, documentsession replay provides a vehicle for reviewing the specifics of aparticular implementation of the execution of the multimodal applicationand specific responses in that implementation provided by a user.

The method of FIG. 5 is implemented with a multimodal browser and amultimodal application operating on a multimodal device supportingmultiple modes of user interaction with the multimodal applicationincluding a voice mode and one or more non-voice modes. The voice modeis represented in this example with audio output through a speaker (177)and audio input through a microphone (176). Non-voice modes arerepresented by user input devices (181), a keyboard and a mouse. In theexample of FIG. 5, the multimodal browser (196) supports a replay modefor document session replay for multimodal applications according toembodiments of the present invention.

Document session replay for multimodal applications may be carried outby iteratively executing the method of FIG. 5. The method of FIG. 5includes identifying (506), by a multimodal browser (196) in dependenceupon a log (504) produced by a Form Interpretation Algorithm (‘FIA’)during a previous document session with a user, a speech prompt (508)provided by a multimodal application in the previous document session.The identified speech prompt may be a recorded audio prompt previouslypresented to a user or the speech prompt may be coded as text andpreviously presented to the user as speech by a text-to-speech engine.

As mentioned above, during the execution of document sessions, an FIAuseful according to embodiments of the present invention maintains a logof time stamped events including speech prompts provided to a user,identifications of X+V pages of the multimodal application presented toa user, responses received from a user, current state frame informationsuch as the values of Voice XML shadow variables, values of DOMelements, as well as other information as will occur to those of skillin the art. The log also may include speech prompts encoded as text forplay by a text to speech engine, pointers to cached recorded audioprompts, text resulting from the recognition of the user's spokenresponses and pointers to the audio recordings of the user's responsesto speech prompts, pointers to cached multimodal web pages, and otherlogged events and pointers to stored information such that the documentsession may be replayed in dependence upon the log. In creating the log,the FIA logs each event invoked during the execution of the documentsession and records, either in the log itself or through a pointer tostored information, the information needed to replay the documentsession. The information needed to replay the document session oftenincludes, for example, text encoding an audio prompt, a pointer to arecorded audio prompt, text resulting from the recognition of the user'sspoken responses and a pointer to a recording of response from a user, apointer to a cached multimodal web page used during the documentsession, and other information that will occur to those of skill in theart.

In the case of recorded audio speech prompts, the log (504) includes apointer to the recorded audio prompt maintained in a data store ofcached recorded audio prompts (552). Such a recorded audio prompt may beidentified in the log (504) and retrieved from the cached recorded audioprompts (552) for replay. Identifying (506) a speech prompt (508)provided by a multimodal application in the previous document session insuch cases may be carried out by traversing the log for identificationsof speech prompts provided to a user during a previous document sessionand retrieving from cached recorded audio prompts (552) the recordedaudio prompt previously played to a user in the previous documentsession. Such identifications of speech prompts included in the log aretime stamped such that the speech prompt and an associated response tothe prompt provided by a user may be correlated.

Speech prompts may also be coded as text. In such cases, the log (504)includes the coded text implementing the audio prompt for replay by atext-to-speech engine. Identifying (506) a speech prompt (508) providedby a multimodal application in the previous document session in suchcases may be carried out by traversing the log for identifications ofspeech prompts provided to a user during a previous document session andretrieving for replay the coded text implementing the speech prompt.Such identifications of speech prompts included in the log are timestamped such that the speech prompt and an associated response to theprompt provided by a user may be correlated.

The method of FIG. 5 also includes identifying (512), by a multimodalbrowser (196) in dependence upon the log (504), a response (514) to theprompt (508) provided by a user of the multimodal application in theprevious document session. Identifying (512) a response (514) to theprompt (508) provided by a user of the multimodal application in theprevious document session may be carried out by traversing the log foridentifications of responses to speech prompts provided by a user duringa previous document session.

As mention above, multimodal applications support response from users inboth voice mode and one or more non-voice modes. During the previousdocument session voice mode responses are recorded and stored fordocument session replay in a data store of cached recorded audioresponses (550). An identification of the recording of the response anda pointer to the recorded audio response in the cache (550) may beincluded in the log such that the recording of the response may beretrieved and replayed. Text resulting from the recognition of theuser's spoken responses during the previous document session is alsostored in the log and is used to recreate the ASR performed during theprevious document session. Non-voice mode responses are recorded as textin the log such that the response may be retrieved from the log andrecreated during document session replay.

The method of FIG. 5 also includes retrieving (520), by the multimodalbrowser (196) in dependence upon the log (504), an X+V page (522) of themultimodal application associated with the speech prompt (508) and theresponse (514). Retrieving (520) an X+V page (522) of the multimodalapplication associated with the speech prompt (508) and the response(514) is carried out by retrieving a X+V page that was cached during theprevious document session in dependence upon an identification of thecached page contained in the log produced by the FIA during the previousdocument session.

The method of FIG. 5 also includes rendering (524), by the multimodalbrowser (196), the visual elements (518) of the retrieved X+V page(522). Rendering (524), by the multimodal browser (196), the visualelements (518) of the retrieved X+V page (522) recreates for the replayuser (502) the visual elements rendered during the speech prompt and theresponse of the previous document session.

The method of FIG. 5 also includes replaying (510), by the multimodalbrowser (196), the speech prompt (508). As mentioned above, the speechprompt may be a recorded audio speech prompt previously played to a userin a previous documents session and stored in a cache of recorded audioprompts or the speech prompt may be encoded as text and stored in thelog for replay by a text to speech engine. Replaying (510) the speechprompt (508) may therefore, in some embodiments, be carried out byreplaying the recorded audio prompt retrieved in dependence upon the logfrom a cache of recorded audio prompts. Alternatively, replaying (510)the speech prompt (508) may be carried out by replaying from the logwith the text to speech engine of the multimodal browser the textencoding speech prompt from the previous document session.

In still another embodiments, the speech prompt may be encoded in thecached X+V page itself. Alternatively, therefore, replaying (510) thespeech prompt may be carried out by replaying the multimodal browser thespeech prompt from the retrieved page of the previous document session.Replaying with the multimodal browser the speech prompt from theretrieved page of the previous document session is carried out byplaying the speech prompt again using a text to speech engine in thesame manner as the speech prompt was presented during the previousdocument session.

The method of FIG. 5 also includes replaying (516), by a multimodalbrowser (196), the response (514). As mentioned above, multimodalapplications support a voice mode and one or more non-voice modes. Theresponse in the method of FIG. 5 may be either a voice mode response ora non-voice mode response. Where the response is a voice mode responsefrom the user, replaying the response according to the method of FIG. 5includes retrieving in dependence upon a pointer or identifier in thelog (504) a recording of the voice mode response from a cache (550) ofrecorded audio responses and playing the recording. Replaying theresponse also includes retrieving from the log the text resulting fromrecognition of the user's spoken response and displaying the text

Where the response is a non-voice mode response, replaying the responseaccording to the method of FIG. 5 includes recreating the non-voice moderesponse. Examples of such non-voice mode responses include mouse clicksand keyboard strokes. Recreating the non-voice mode response may becarried out by executing a Javascript against DOM.

As will occur to those of skill in the art, the method of FIG. 5provides document session replay for multimodal applications that may beused in a number of fields of endeavor. Once such endeavor is thecreation of a legal document for a user by a multimodal application. Insuch a use case, a multimodal application may prompt users forinformation needed to create the legal document and store the recordedresponses. Using the log, the document session may then be replayed toconfirm the answers provided by the user, resolve disputes arising fromthe creation of the legal document, or for any other reason that willoccur to those of skill in the art.

The example provided of creating a legal document is for explanation andnot for limitation. In fact, the creation of legal documents are onlyone of many fields that may benefit from document session replayaccording to the present invention. Document session replay according tothe present invention is meant for broad use in a wide variety ofcontexts and all such uses and contexts are well within the scope of thepresent invention.

Document session replay for multimodal applications according toembodiments of the present invention does not require the documentsession replay to start at the beginning of the previous documentsession. That is, replay users may wish to start document session replaynear specific content of interest. Such specific content of interest mayinclude particular keywords in responses from users, particular speechprompts presented to users in the previous document session, particularresponses to speech prompts in the previous document session, particularrendered visual elements in the previous document session and so on aswill occur to those of skill in the art. Document session replay formultimodal applications according to the present invention may thereforeinclude features for initiating replay near such content of interest.

For further explanation, FIG. 6 sets forth additional aspects of someembodiments of document session replay for multimodal applicationsaccording to the present invention. The method of FIG. 6 includesreceiving (602) from a user (502) a query (604) for content of interestin the previous document session. Receiving (602) from a user (502) aquery (604) for content of interest in the previous document session maybe carried out through a GUI provided by the multimodal browser designedto accept from a user a query for content of interest. Such a query mayinclude text such keywords, SQL queries or any other query as will occurto those of skill in the art

The method of FIG. 6 includes identifying (606) content of interest(610) in dependence upon the log (608). Identifying (606) content ofinterest (610) in dependence upon the log (608) may be carried out bysearching the log against terms of the query.

The method of FIG. 6 also includes initiating (612) replay of previousdocument session near the content of interest (610) includinginitializing (614) the application state of the multimodal application(195) for replay near the content of interest. Initializing (614) theapplication state of the multimodal application (195) for replay nearthe content of interest includes setting the current state from of theapplication by setting the values of, for example, of voice XML shadowvariables and values of DOM elements such that the document sessionreplay may make use of the multimodal application itself.

Exemplary embodiments of the present invention are described largely inthe context of a fully functional computer system for document sessionreplay for multimodal applications. Readers of skill in the art willrecognize, however, that the present invention also may be embodied in acomputer program product disposed on computer-readable signal bearingmedia for use with any suitable data processing system. Such signalbearing media may be transmission media or recordable media formachine-readable information, including magnetic media, optical media,or other suitable media. Examples of recordable media include magneticdisks in hard drives or diskettes, compact disks for optical drives,magnetic tape, and others as will occur to those of skill in the art.Examples of transmission media include telephone networks for voicecommunications and digital data communications networks such as, forexample, Ethernets™ and networks that communicate with the InternetProtocol and the World Wide Web. Persons skilled in the art willimmediately recognize that any computer system having suitableprogramming means will be capable of executing the steps of the methodof the invention as embodied in a program product. Persons skilled inthe art will recognize immediately that, although some of the exemplaryembodiments described in this specification are oriented to softwareinstalled and executing on computer hardware, nevertheless, alternativeembodiments implemented as firmware or as hardware are well within thescope of the present invention.

It will be understood from the foregoing description that modificationsand changes may be made in various embodiments of the present inventionwithout departing from its true spirit. The descriptions in thisspecification are for purposes of illustration only and are not to beconstrued in a limiting sense. The scope of the present invention islimited only by the language of the following claims.

1. A method for document session replay for multimodal applications, themethod implemented with a multimodal browser and a multimodalapplication operating on a multimodal device supporting multiple modesof user interaction with the multimodal application, the modes of userinteraction including a voice mode and one or more non-voice modes, themethod comprising: identifying, by a multimodal browser in dependenceupon a log produced by a Form Interpretation Algorithm (‘FIA’) during aprevious document session with a user, a speech prompt provided by amultimodal application in the previous document session; identifying, bya multimodal browser in dependence upon the log, a response to theprompt provided by a user of the multimodal application in the previousdocument session; retrieving, by the multimodal browser in dependenceupon the log, an X+V page of the multimodal application associated withthe speech prompt and the response; rendering, by the multimodalbrowser, the visual elements of the retrieved X+V page on a display;replaying, by the multimodal browser, the speech prompt via at least onespeaker; and replaying, by a multimodal browser, the response via atleast one speaker.
 2. The method of claim 1 wherein replaying the speechprompt further comprises: replaying from the log with the multimodalbrowser the speech prompt from the previous document session.
 3. Themethod of claim 1 wherein replaying the prompt further comprises:replaying with the multimodal browser the speech prompt from theretrieved page of the previous document session.
 4. The method of claim1 wherein the response comprises a voice mode response from the user;and replaying the response further comprises retrieving a recording ofthe voice mode response and playing the recording.
 5. The method ofclaim 1 wherein the response comprises a non-voice mode response; andreplaying the response further comprises recreating the non-voice moderesponse.
 6. The method of claim 1 further comprising: receiving from auser a query for content of interest in the previous document session.7. The method of claim 6 further comprising: initiating replay ofprevious document session near the content of interest includinginitializing the application state of the multimodal application forreplay near the content of interest.
 8. An apparatus for documentsession replay for multimodal applications, the apparatus including amultimodal browser and a multimodal application operating on amultimodal device supporting multiple modes of user interaction with themultimodal application, the modes of user interaction including a voicemode and one or more non-voice modes, the apparatus comprising acomputer processor and a computer memory operatively coupled to thecomputer processor, the computer memory having disposed within itcomputer program instructions capable of: identifying, in dependenceupon a log produced by a Form Interpretation Algorithm (‘FIA’) during aprevious document session with a user, a speech prompt provided by amultimodal application in the previous document session; identifying, independence upon the log, a response to the prompt provided by a user ofthe multimodal application in the previous document session; retrieving,in dependence upon the log, an X+V page of the multimodal applicationassociated with the speech prompt and the response; rendering, thevisual elements of the retrieved X+V page on a display; replaying thespeech prompt via at least one speaker; and replaying the response viaat least one speaker.
 9. The apparatus of claim 8 wherein computerprogram instructions capable of replaying the speech prompt furthercomprises computer program instructions capable of: replaying from thelog with the multimodal browser the speech prompt from the previousdocument session.
 10. The apparatus of claim 8 wherein computer programinstructions capable of replaying the prompt further comprises computerprogram instructions capable of: replaying with the multimodal browserthe speech prompt from the retrieved page of the previous documentsession.
 11. The apparatus of claim 8 wherein the response comprises avoice mode response from the user; and computer program instructionscapable of replaying the response further comprise computer programinstructions capable of: retrieving a recording of the voice moderesponse; and playing the recording.
 12. The apparatus of claim 8wherein the response comprises a non-voice mode response; and computerprogram instructions capable of replaying the response further comprisescomputer program instructions capable of recreating the non-voice moderesponse.
 13. The apparatus of claim 8 wherein the computer memory alsohas disposed within it computer program instructions capable ofreceiving from a user a query for content of interest in the previousdocument session.
 14. The apparatus of claim 13 wherein the computermemory also has disposed within it computer program instructions capableof initiating replay of previous document session near the content ofinterest including initializing the application state of the multimodalapplication for replay near the content of interest.
 15. A computerprogram product for document session replay for multimodal applications,the computer program product including a multimodal applicationoperating on a multimodal device supporting multiple modes of userinteraction with the multimodal application, the modes of userinteraction including a voice mode and one or more non-voice modes, thecomputer program product disposed upon a computer-readable recordingmedium, the computer program product comprising computer programinstructions capable of: identifying, in dependence upon a log producedby a Form Interpretation Algorithm (‘FIA’) during a previous documentsession with a user, a speech prompt provided by a multimodalapplication in the previous document session; identifying, in dependenceupon the log, a response to the prompt provided by a user of themultimodal application in the previous document session; retrieving, independence upon the log, an X+V page of the multimodal applicationassociated with the speech prompt and the response; rendering, thevisual elements of the retrieved X+V page on a display; replaying thespeech prompt via at least one speaker; and replaying the response viaat least one speaker.
 16. The computer program product of claim 15wherein the response comprises a voice mode response from the user; andcomputer program instructions capable of replaying the response furthercomprise computer program instructions capable of: retrieving arecording of the voice mode response; and playing the recording.
 17. Thecomputer program product of claim 15 wherein the computer memory alsohas disposed within it computer program instructions capable ofreceiving from a user a query for content of interest in the previousdocument session.
 18. The computer program product of claim 17 whereinthe computer memory also has disposed within it computer programinstructions capable of initiating replay of previous document sessionnear the content of interest including initializing the applicationstate of the multimodal application for replay near the content ofinterest.